Skip to content

fix(posting): make rollup failures diagnosable in logs - #9808

Merged
matthewmcneely merged 1 commit into
mainfrom
fix/rollup-error-logging
Aug 6, 2026
Merged

fix(posting): make rollup failures diagnosable in logs#9808
matthewmcneely merged 1 commit into
mainfrom
fix/rollup-error-logging

Conversation

@matthewmcneely

@matthewmcneely matthewmcneely commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Description

Two problems make a failing rollup nearly impossible to triage from logs. Both
turned up while looking at #9794, where the reporter's log line was truncated
right before the useful part.

1. Reversed format arguments in incrRollupi.Process

glog.Warningf("Error rolling up key [%v]: %v", err, key)

err and key are swapped, so the message reads Error rolling up key [<the error>]: <the key>. The key also rendered as a decimal byte slice, which can't
be pasted into dgraph debug --lookup. Now %x with the arguments in the right
order.

2. The pIterator.seek error message dumps the whole posting list

List.iterate formatted l.Print() into its error, and Print() dumps the
entire posting list plus mutation map with posting values included. On a
multi-part list holding large values that's a multi-megabyte log line. Since
ReadPostingList re-queues a key for rollup on every read while deltas remain,
and doRollup retries a given key roughly every 10 seconds, the same giant line
repeats for as long as the key stays broken.

Worse for triage: errors.Wrapf appends the cause after the format string, so
the actual error is the first thing to get cut off. seek has only two failure
modes and they have completely different root causes, so losing that suffix
loses the diagnosis:

deleteBelowTs (%d) must be greater than the minTs in the list (%d)
cannot read initial list part for list with base key %s

This PR adds List.debugString(), which reports the key (hex), minTs,
maxTs, splits, and mutation-layer counts, with no posting values, and uses it
on the three seek error paths in iterate and findPostingWithItr. The two
paths in findPostingWithItr were formatting mutationMap.print(), which has
the same unbounded-value problem.

Print() is unchanged and stays available for interactive debugging.

Sample of the new output shape:

cannot initialize iterator when calling List.iterate key: 000764617461..., minTs: 58256931,
maxTs: 58550095, splits: [1 8523883114575535643], mutationMap: {committed: 1, proposed: false,
deleteAllMarker: 18446744073709551615}: could not read list part with key ...: Key not found

Logging only. No behavior change.

Refs #9794.

Checklist

  • The PR title follows the
    Conventional Commits syntax, leading
    with fix:, feat:, chore:, ci:, etc.
  • Code compiles correctly and linting (via trunk) passes locally

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

Two problems make a failing rollup nearly impossible to triage from logs, both
surfaced by #9794.

The format arguments in incrRollupi.Process are reversed, so the message reads
"Error rolling up key [<the error>]: <the key>". The key also printed as a
decimal byte slice rather than hex, which can't be fed to `dgraph debug
--lookup`.

More importantly, the error returned when pIterator.seek fails formats
List.Print() into the message. Print() dumps the entire posting list and
mutation map, posting values included. On a multi-part list holding large
values that's a multi-megabyte log line, and since ReadPostingList re-queues a
key for rollup whenever deltas remain, the same line repeats every few seconds
for as long as the key stays broken. The actual cause is appended after the
dump by errors.Wrapf, so it's the first thing to get truncated, which is
exactly what happened in #9794.

Add List.debugString(), which reports the key, minTs, maxTs, splits, and
mutation-layer counts without any values, and use it on the three seek error
paths. Print() stays for interactive debugging.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@matthewmcneely
matthewmcneely requested a review from a team as a code owner August 5, 2026 17:59
@matthewmcneely
matthewmcneely merged commit 64804bd into main Aug 6, 2026
19 checks passed
@matthewmcneely
matthewmcneely deleted the fix/rollup-error-logging branch August 6, 2026 16:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants